\subsection{UPC Parallel I/O {\tt <upc\_io.h>}}
\index{\_\_UPC\_IO\_\_}
\index{upc\_io.h}

\npf This subsection provides the UPC parallel extensions of [ISO/IEC00 
    Sec 7.19].  All the characteristics of library functions described
    in [ISO/IEC00 Sec 7.1.4] apply to these as well.  Implementations
    that support this interface shall predefine the feature macro {\tt
    \_\_UPC\_IO\_\_} to the value 1.

{\bf Common Constraints}

\index{upc\_flag\_t}
\np All UPC-IO functions are collective and must be called by all threads collectively.\footnote{Note that 
collective does not necessarily imply barrier synchronization.  The synchronization behavior of the 
UPC-IO data movement library functions is explicitly controlled by using the {\tt
flags} flag argument. See Section \ref{upc-flag-t} for details.}

\np If a program calls {\tt exit}, {\tt upc\_global\_exit}, or returns from {\tt main} with a
UPC file still open, the file will automatically be closed at program
termination, and the effect will be equivalent to {\tt upc\_all\_fclose} being
implicitly called on the file.

\index{end of file}
\np If a program attempts to read past the end of a file, the read function
will read data up to the end of file and return the number of bytes actually
read, which may be less than the amount requested.

\np Writing past the end of a file increases the file size.

\np If a program seeks to a location past the end of a file and writes
starting from that location, the data in the intermediate (unwritten)
portion of the file is undefined. For example, if a program opens a new file
(of size 0 bytes), seeks to offset 1024 and writes some data beginning from
that offset, the data at offsets 0--1023 is undefined. Seeking past the end
of file and performing a write causes the current file size to immediately
be extended up to the end of the write. However, just seeking past the end
of file or attempting to read past the end of file, without a write, does
not extend the file size.

\np All generic pointers-to-shared passed to the I/O functions (as function
arguments or indirectly through the list I/O arguments) are treated as if
they had a phase field of zero (that is, the input phase is ignored).

\np All UPC-IO read/write functions take an argument {\tt flags}
    of type {\tt upc\_flag\_t}. The semantics of this argument is
    defined in Section \ref{upc-flag-t}. These semantics apply only to
    memory locations in user-provided buffers, not to the read/write
    operations on the storage medium or any buffer memory internal to
    the library implementation.


\np The {\tt flags} flag is included even on the fread/fwrite\_local functions
(which take a pointer-to-local as the buffer argument) in order to provide
well-defined semantics for the case where one or more of the pointer-to-local
arguments references a shared object (with local affinity). In the case where
all of the pointer-to-local arguments in a given call reference only private
objects, the {\tt flags} flag provides no useful additional guarantees and is
recommended to be passed as {\tt UPC\_IN\_NOSYNC|UPC\_OUT\_NOSYNC} to maximize performance.

\np The arguments to all UPC-IO functions are single-valued except where 
    explicitly noted in the function description.

\index{file consistency}
\index{file atomicity}
\np UPC-IO, by default, supports weak consistency and atomicity semantics. The
default (weak) semantics are as follows. The data written to a file by a thread is
only guaranteed to be visible to another thread after all threads have
collectively closed or synchronized the file.

\np Writes to a file from a given thread are always guaranteed to be visible to
subsequent file reads by the \textit{same} thread, even without an intervening
call to collectively close or synchronize the file.

\np Byte-level data consistency is supported.

\np If concurrent writes from multiple threads overlap in the file, the
resulting data in the overlapping region is undefined with the weak
consistency and atomicity semantics

\np When reading data being concurrently written by another thread, the data
that gets read is undefined with the weak consistency and atomicity
semantics.

\np File reads into overlapping locations in a shared buffer in memory using individual file
pointers or list I/O functions leads to undefined data in the target buffer under the
weak consistency and atomicity semantics.

\np A given file must not be opened at same time by the POSIX I/O
and UPC-IO libraries.

\np Except where otherwise noted, all UPC-IO functions return
NON-single-valued errors; that is, the occurrence of an error need only be
reported to at least one thread, and the {\tt errno} value reported to each such
thread may differ. When an error is reported to ANY thread, the position of
ALL file pointers for the relevant file handle becomes undefined.

\np The error values that UPC-IO functions may set in {\tt errno} are
implementation-defined, however the {\tt perror()} and {\tt strerror()} functions are still
guaranteed to work properly with {\tt errno} values generated by UPC-IO.

\np UPC-IO functions can not be called between {\tt upc\_notify} and corresponding {\tt upc\_wait} statements.

\subsubsection{Background}

\paragraph{File Accessing and File Pointers}

%\begin{figure}[h]
%\includegraphics[bb=0 0 620 339, width=1\textwidth]{figure1.eps}
%\includegraphics[width=1\textwidth]{figure1}
%\caption{UPC-IO File Access Methods}
%\end{figure}

\index{file pointer}
\index{common file pointer}
\index{individual file pointer}

\npf Collective UPC-IO accesses can be done in and out of shared and private
buffers, thus local and shared reads and writes are generally supported.
In each of these cases, file pointers could be either common or individual.
Note that in UPC-IO, common file pointers cannot be used in conjunction with
pointer-to-local buffers. 
File pointer modes are specified by passing a flag to the collective
{\tt upc\_all\_fopen} function and can be changed using {\tt upc\_all\_fcntl}.
When a file is opened with the common file pointer flag, all threads share a
common file pointer. When a file is opened with the individual file pointer
flag, each thread gets its own file pointer.

\np UPC-IO also provides file-pointer-independent list file accesses by specifying
explicit offsets and sizes of data that is to be accessed. List IO can also
be used with either pointer-to-local buffers or pointer-to-shared buffers.

\np Examples 1-3 and their associated figures, Figures 2-4, give typical
instances of UPC-IO usage. Error checking is omitted for brevity.

\np EXAMPLE 1: collective read operation using individual file pointers
\begin{verbatim}
#include <upc.h>
#include <upc_io.h>
double buffer[10];  // and assuming a total of 4 THREADS
upc_file_t *fd; 

fd = upc_all_fopen( "file", UPC_RDONLY | UPC_INDIVIDUAL_FP, 0, NULL );
upc_all_fseek( fd, 5*MYTHREAD*sizeof(double), UPC_SEEK_SET );
upc_all_fread_local( fd, buffer, sizeof(double), 10, 
                          UPC_IN_ALLSYNC | UPC_OUT_ALLSYNC);
upc_all_fclose(fd);
\end{verbatim}

%\begin{figure}[h]
%\includegraphics[bb=0 0 630 314, width=1\textwidth]{figure2.eps}
%\includegraphics[width=1\textwidth]{figure2}
%\caption{Collective read into private buffers can provide a canonical file-view}
%\end{figure}

 Each thread reads a block of data into a private buffer from a particular thread-specific offset.

\np EXAMPLE 2: a collective read operation using a common file pointer. 
The data read is stored into a shared buffer,
having a block size of 5 elements. The user selects the type of file pointer
at file-open time. The user can select either individual file pointers by
passing the flag {\tt UPC\_INDIVIDUAL\_FP} to the function {\tt upc\_all\_fopen}, or the
common file pointer by passing the flag {\tt UPC\_COMMON\_FP} to {\tt upc\_all\_fopen}.

\begin{verbatim}
#include <upc.h>
#include <upc_io.h>
shared [5] float buffer[20];  // and assuming a total of 4 static THREADS
upc_file_t *fd; 

fd = upc_all_fopen( "file", UPC_RDONLY | UPC_COMMON_FP, 0, NULL );
upc_all_fread_shared( fd, buffer, upc_blocksizeof(buffer), 
     upc_elemsizeof(buffer), 20, UPC_IN_ALLSYNC | UPC_OUT_ALLSYNC);
/* or equivalently: 
 *  upc_all_fread_shared( fd, buffer, 5, sizeof(float), 20, 
                                 UPC_IN_ALLSYNC | UPC_OUT_ALLSYNC); 
 */
\end{verbatim}
%\begin{figure}[h]
%\includegraphics[bb=0 0 604 146, width=1\textwidth]{figure3.eps}
%\includegraphics[width=1\textwidth]{figure3}
%\caption{Collective read into a blocked shared buffer can provide a partitioned file-view}
%\end{figure}

\paragraph{Synchronous and Asynchronous I/O}

\index{asynchronous I/O}
\npf I/O operations can be synchronous (blocking) or asynchronous (non-blocking).
While synchronous calls are quite simple and easy to use from a programming
point of view, asynchronous operations allow the overlapping of computation
and I/O to achieve improved performance. Synchronous calls block and wait
until the corresponding I/O operation is completed. On the other hand, an
asynchronous call starts an I/O operation and returns immediately. Thus, the
executing process can turn its attention to other processing needs while the
I/O is progressing.

\index{upc\_all\_ftest\_async}
\index{upc\_all\_fwait\_async}
\np UPC-IO supports both synchronous and asynchronous I/O functionality. The
asynchronous I/O functions have the same syntax and basic semantics as their synchronous
counterparts, with the addition of the {\tt async} suffix in their names. The
asynchronous I/O functions have the restriction that only one (collective)
asynchronous operation can be active at a time on a given file handle. That
is, an asynchronous I/O function must be completed by calling
{\tt upc\_all\_ftest\_async} or {\tt upc\_all\_fwait\_async} before another asynchronous
I/O function can be called on the same file handle. This restriction is
similar to the restriction MPI-IO [MPI2] has on split-collective I/O functions:
only one split collective operation can be outstanding on an MPI-IO file
handle at any time.

\paragraph{Consistency and Atomicity Semantics}

\index{file consistency}
\index{file atomicity}
\npf The consistency semantics define when the data written
to a file by a thread is visible to other threads. The atomicity semantics
define the outcome of operations in which multiple threads write
concurrently to a file or shared buffer and some of the writes overlap each other. For
performance reasons, UPC-IO uses weak consistency and
atomicity semantics by default. The user can select stronger semantics either by
opening the file with the flag {\tt UPC\_STRONG\_CA} or by calling 
{\tt upc\_all\_fcntl} with the command 
{\tt UPC\_SET\_STRONG\_CA\_SEMANTICS}.

\index{upc\_all\_fsync}
\np The default (weak) semantics are as follows. The data written by a thread is
only guaranteed to be visible to another thread after all threads have
called {\tt upc\_all\_fclose} or {\tt upc\_all\_fsync}. (Note that the data \textit{%
may} be visible to other threads before the call to {\tt upc\_all\_fclose} or
{\tt upc\_all\_fsync} and that the data may become visible to different
threads at different times.) Writes from a given thread are always
guaranteed to be visible to subsequent reads by the \textit{same} thread,
even without an intervening call to {\tt upc\_all\_fclose} or {\tt upc\_all\_fsync}.
Byte-level data consistency is supported. So for example, if thread 0
writes one byte at offset 0 in the file and thread 1 writes one byte at
offset 1 in the file, the data from both threads will get written to the
file. If concurrent writes from multiple threads overlap in the file, the
resulting data in the overlapping region is undefined. Similarly, if one
thread tries to read the data being concurrently written by another thread,
the data that gets read is undefined. Concurrent in this context means any
two read/write operations to the same file handle with no intervening calls
to {\tt upc\_all\_fsync} or {\tt upc\_all\_fclose}.

\np For the functions that read into or write from a shared buffer using a
common file pointer, the weak consistency semantics are defined as follows.
Each call to {\tt upc\_all\_\{fread,fwrite\}\_shared[\_async]} with a common file
pointer behaves as if the read/write operations were performed by a single,
distinct, anonymous thread which is different from any compute thread (and
different for each operation). In other words, NO file reads are guaranteed to
see the results of file writes using the common file pointer until after a close
or sync under the default weak consistency semantics.

\index{UPC\_STRONG\_CA}
\index{UPC\_SET\_STRONG\_CA\_SEMANTICS}
\np By passing the {\tt UPC\_STRONG\_CA} flag to {\tt upc\_all\_fopen} or 
by calling {\tt upc\_all\_fcntl} with the command 
{\tt UPC\_SET\_STRONG\_CA\_SEMANTICS}, the user selects strong
consistency and atomicity semantics. In this case, the data written by a
thread is visible to other threads as soon as the file write on the calling
thread returns. In the case of writes from multiple threads to overlapping
regions in the file, the result would be as if the individual write function
from each thread occurred atomically in some (unspecified) order.
Overlapping writes to a file in a single (list I/O) write function on a
single thread are not permitted (see Section \ref{io-func-list}). While strong consistency
and atomicity semantics are selected on a given file handle, the {\tt
flags} argument to all fread/fwrite functions on that handle is ignored
and always treated as {\tt UPC\_IN\_ALLSYNC | UPC\_OUT\_ALLSYNC}.

\np The consistency semantics also define the outcome in the case of overlapping
reads into a shared buffer in memory when using individual file pointers or
list I/O functions. By default, the data in the overlapping space is
undefined. If the user selects strong consistency and atomicity, the result would be as
if the individual read functions from each thread occurred atomically in
some (unspecified) order. Overlapping reads into memory buffers in a single
(list I/O) read function on a single thread are not permitted (see Section
\ref{io-func-list}).

\np Note that in strong consistency and atomicity mode, atomicity is guaranteed at the UPC-IO function
level. The entire operation specified by a single function is performed
atomically, regardless of whether it represents a single, contiguous
read/write or multiple noncontiguous reads or writes as in a list I/O
function.

\np EXAMPLE 1:  three threads write data to a file
concurrently, each with a single list I/O function. The numbers indicate
file offsets and brackets indicate the boundaries of a listed vector. Each
thread writes its own thread id as the data values:

\begin{verbatim}
thread 0:    {1  2  3}   {5  6  7  8}
thread 1: {0  1  2}{3  4  5}
thread 2:             {4  5  6}   {8  9 10 11}
\end{verbatim}

\np With the default weak semantics, the results in the overlapping locations
are undefined. Therefore, the result in the file would be the following,
where x represents undefined data.

\begin{verbatim}
File:     1  x   x  x  x  x  x  0  x  2  2  2
\end{verbatim}
\np That is, the data from thread 1 is written at location 0, the data from
thread 0 is written at location 7, and the data from thread 2 is written at
locations 9, 10, and 11, because none of these locations had overlapping
writes. All other locations had overlapping writes, and consequently, the
result at those locations is undefined.

\np If the file were opened with the {\tt UPC\_STRONG\_CA} flag, strong
consistency and atomicity semantics would be
in effect. The result, then, would depend on the order in which the writes
from the three threads actually occurred. Since six different orderings are
possible, there can be six outcomes. Let us assume, for example, that the
ordering was the write from thread 0, followed by the write from thread 2,
and then the write from thread 1. The (list I/O) write from each thread
happens atomically. Therefore, for this ordering, the result would be:

\begin{verbatim}
File:     1  1  1  1  1  1  2  0  2  2  2  2
\end{verbatim}

\np We note that if instead of using a single list I/O function each thread used
a separate function to write each contiguous portion, there would be six
write functions, two from each thread, and the atomicity would be at the
granularity of the write operation specified by each of those functions.

\paragraph{File Interoperability}

\index{file interoperability}
\npf UPC-IO does not specify how an implementation may store the data in a file
on the storage device. Accordingly, it is implementation-defined whether or not 
a file created by UPC-IO can be directly accessed by using C/POSIX I/O functions.
However, the UPC-IO implementation must specify how
the user can retrieve the file from the storage system as a linear sequence
of bytes and vice versa. Similarly, the implementation must specify how
familiar operations, such as the equivalent of POSIX {\tt ls}, {\tt cp}, {\tt rm}, and {\tt mv} can
be performed on the file.

\subsubsection{Predefined Types}

\index{upc\_off\_t}
\npf The following types are defined in {\tt <upc\_io.h>}

\np{\tt upc\_off\_t} is a signed integral type that is capable of representing the size of the largest
file supported by the implementation.

\index{upc\_file\_t}
\np{\tt upc\_file\_t} is an opaque shared data type of incomplete type (as defined in [ISO/IEC00 Sec 6.2.5]) that represents an open file handle.

\np{\tt upc\_file\_t} objects are always manipulated via a pointer (that is,
{\tt upc\_file\_t *}).

\np{\tt upc\_file\_t} is a shared data type. It is allowed to pass a ({\tt upc\_file\_t
*}) across threads, and two pointers to {\tt upc\_file\_t} that reference the same
logical file handle will always compare equal.

{\bf Advice to implementors}

\np The definition of {\tt upc\_file\_t} does not restrict the implementation to
store all its metadata with affinity to one thread. Each thread can still
have local access to its metadata. For example, below is a simple approach
an implementation could use:

\begin{verbatim}
#include <upc.h>
#include <upc_io.h>
/* for a POSIX-based implementation */
typedef int my_internal_filehandle_t; 

#ifdef _UPC_INTERNAL
  typedef struct _local_upc_file_t  {
    my_internal_filehandle_t fd;
    ... other metadata ...
  } local_upc_file_t;
#else
  struct _local_upc_file_t;
#endif

typedef shared struct _local_upc_file_t upc_file_t;

upc_file_t *upc_all_fopen(...) {

   upc_file_t *handles = 
            upc_all_alloc(THREADS, sizeof(upc_file_t));

   /* get my handle */
   upc_file_t *myhandle = &(handles[MYTHREAD]); 
   
   /* cast to a pointer-to-local */
   local_upc_file_t* mylocalhandle = (local_upc_file_t*)myhandle;

   /* setup my metadata using pointer-to-local */
   mylocalhandle->fd = open(...);

   ...

   return handles;
}
\end{verbatim}

\np The basic idea is that the ``handle'' exposed to the user actually points to
a cyclic, distributed array. As a result, each thread has easy, local access
to its own internal handle metadata with no communication, while maintaining
the property that the handle that UPC-IO exposes to the client is a
single-valued pointer-to-shared. An additional advantage is that a thread
can directly access the metadata for other threads, which may occasionally
be desirable in the implementation.

\subsubsection{UPC File Operations}

{\bf Common Constraints}
\label{upc-io-common}

\index{file pointer}
\index{common file pointer}
\index{individual file pointer}
\npf When a file is opened with an individual file pointer, each thread will get
its own file pointer and advance through the file at its own pace.

\np When a common file pointer is used, all threads positioned in the file
will be aligned with that common file pointer.

\np Common file pointers cannot be used in conjunction with pointers-to-local 
(and hence cannot operate on private objects).

\np No function in this section may be called while an asynchronous
operation is pending on the file handle, except where otherwise noted.
% it is otherwise noted for upc_all_fcntl

\paragraph{The {\tt upc\_all\_fopen} function}
\label{io-func-open}

{\bf Synopsis}
\index{file open}
\index{upc\_all\_fopen}
\index{UPC\_RDONLY}
\index{UPC\_WRONLY}
\index{UPC\_RDWR}
\index{UPC\_COMMON\_FP}
\index{UPC\_INDIVIDUAL\_FP}
\index{UPC\_APPEND}
\index{UPC\_CREATE}
\index{UPC\_EXCL}
\index{UPC\_STRONG\_CA}
\index{UPC\_TRUNC}
\index{UPC\_DELETE\_ON\_CLOSE}

\npf\vspace{-2.5em}
 \begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    upc_file_t *upc_all_fopen(const char *fname, int flags, 
         size_t numhints, struct upc_hint const *hints);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fopen} opens the file identified by the filename {\tt fname} for
input/output operations.

\np The flags parameter specifies the access mode. The valid flags and their
meanings are listed below. Of these flags, exactly one of {\tt UPC\_RDONLY},
{\tt UPC\_WRONLY}, or {\tt UPC\_RDWR}, and one of {\tt UPC\_COMMON\_FP} or {\tt UPC\_INDIVIDUAL\_FP},
must be used. Other flags are optional. Multiple flags can be combined by
using the bitwise OR operator ({\textbar}), and each flag has a unique
bitwise representation that can be unambiguously tested using the bitwise
AND operator(\&).
\begin{description}
\item{\tt UPC\_RDONLY}
Open the file in read-only mode
\item{\tt UPC\_WRONLY}
Open the file in write-only mode
\item{\tt UPC\_RDWR}
Open the file in read/write mode
\item{\tt UPC\_INDIVIDUAL\_FP}
Use an individual file pointer for all file accesses (other than 
list I/O)
\item{\tt UPC\_COMMON\_FP}
Use the common file pointer for all file accesses (other than 
list I/O)
\item{\tt UPC\_APPEND}
Set the \textit{initial} position of the file pointer to end of file. 
(The file pointer is not moved to the end of file after each 
read/write)
\item{\tt UPC\_CREATE}
Create the file if it does not already exist. If the named file does not exist and this flag is not passed, the function fails with an error.
\item{\tt UPC\_EXCL}
Must be used in conjunction with {\tt UPC\_CREATE}. The open will fail if the 
 file already exists.
\item{\tt UPC\_STRONG\_CA}
Set strong consistency and atomicity semantics
\item{\tt UPC\_TRUNC}
Open the file and truncate it to zero length.  The file must be opened before
writing.
\item{\tt UPC\_DELETE\_ON\_CLOSE} 
Delete the file automatically on close
\end{description}

\np The {\tt UPC\_COMMON\_FP} flag specifies that all accesses (except for the list
I/O operations) will use the common file pointer. The {\tt UPC\_INDIVIDUAL\_FP} flag
specifies that all accesses will use individual file pointers (except for the
list I/O operations). Either {\tt UPC\_COMMON\_FP} or {\tt UPC\_INDIVIDUAL\_FP} must be
specified or {\tt upc\_all\_fopen} will return an error.

\np The {\tt UPC\_STRONG\_CA} flag specifies strong consistency and atomicity semantics.
The data written by a thread is visible to other threads as soon as the
write on the calling thread returns. In the case of writes from multiple
threads to overlapping regions in the file, the result would be as if the
individual write function from each thread occurred atomically in some
(unspecified) order. In the case of overlapping reads into a shared buffer
in memory when using individual file pointers or list I/O functions, the result
would be as if the individual read functions from each thread occurred
atomically in some (unspecified) order.

\np If the flag {\tt UPC\_STRONG\_CA} is not specified, weak semantics are provided. The
data written by a thread is only guaranteed to be visible to another thread
after all threads have called {\tt upc\_all\_fclose} or {\tt upc\_all\_fsync}. (Note
that the data \textit{may} be visible to other threads before the call to
{\tt upc\_all\_fclose} or {\tt upc\_all\_fsync} and that the data may become visible
to different threads at different times.) Writes from a given thread are
always guaranteed to be visible to subsequent reads by the \textit{same}
thread, even without an intervening call to {\tt upc\_all\_fclose} or
{\tt upc\_all\_fsync}. Byte-level data consistency is supported. For the
purposes of atomicity and consistency semantics, each call to
{\tt upc\_all\_\{fread,fwrite\}\_shared[\_async]} with a common file pointer behaves
as if the read/write operations were performed by a single, distinct,
anonymous thread which is different from any compute thread (and different
for each operation).''\footnote{%
In other words, NO reads are guaranteed to see the results of writes using
the common file pointer until after a close or sync when {\tt UPC\_STRONG\_CA} is not
specified.}

\index{upc\_hint}
\index{file hints}
\np Hints can be passed to the UPC-IO library as an array of key-value pairs%
\footnote{%
The contents of the key/value pairs passed by all the threads must be single-valued.} 
of strings. {\tt numhints} specifies the number of hints in the {\tt hints}
array; if {\tt numhints} is zero, the {\tt hints} pointer is ignored. The user can free
the {\tt hints} array and associated character strings as soon as the open call returns.  The following type is defined in {\tt <upc\_io.h>}:

\begin{verbatim}
    struct upc_hint
\end{verbatim} 
holds each element of the {\tt hints} array and contain at least the following initial members, in this order.

\begin{verbatim}
    const char *key;
    const char *value;
\end{verbatim}

\np{\tt upc\_all\_fopen} defines a number hints.
An implementation is free to 
support additional hints. An implementation is free to ignore any hint
provided by the user. Implementations should \textit{silently} ignore any
hints they do not support or recognize. The predefined hints and their
meanings are defined below. An implementation is not required to interpret
these hint keys, but if it does interpret the hint key, it must provide
the functionality described.  All hints are single-valued character strings
(the content is single-valued, not the location).

\begin{description}
\item [{\tt access\_style}] (comma-separated list of strings): indicates the manner in
which the file is expected to be accessed. The hint value is a
comma-separated list of any the following: ``read\_once", ``write\_once",
``read\_mostly", ``write\_mostly", ``sequential", and ``random". Passing such a hint
does not place any constraints on how the file may actually be accessed by
the program, although accessing the file in a way that is different from the
specified hint may result in lower performance. 
\item [{\tt collective\_buffering}] (boolean): specifies whether the application may
benefit from collective buffering optimizations. Allowed values for this key
are ``true'' and ``false''. Collective buffering parameters can be further
directed via additional hints: cb\_buffer\_size, and cb\_nodes. 
\item [{\tt cb\_buffer\_size}] (decimal integer): specifies the total buffer space that the
implementation can use on each thread for collective buffering. 
\item [{\tt cb\_nodes}] (decimal integer): specifies the number of target threads or I/O nodes to
be used for collective buffering. 
\item [{\tt file\_perm}] (string): specifies the file permissions to use for file
creation. The set of allowed values for this key is implementation defined.

\item [{\tt io\_node\_list}] (comma separated list of strings): specifies the list of I/O
devices that should be used to store the file and is only relevant when the
file is created. 
\item [{\tt nb\_proc}] (decimal integer): specifies the number of threads that will typically be
used to run programs that access this file and is only relevant when the
file is created.
\item[{\tt striping\_factor}] (decimal integer): specifies the number of I/O devices that the
file should be striped across and is relevant only when the file is created.
\item[{\tt start\_io\_device}] (decimal integer): specifies the number of the first I/O device
from which to start striping the file and is relevant only when the file is
created.
\item[{\tt striping\_unit}] (decimal integer): specifies the striping unit to be used for the
file. The striping unit is the amount of consecutive data assigned to one
I/O device before progressing to the next device, when striping across a
number of devices. It is expressed in bytes. This hint is relevant only when
the file is created.
\end{description}

\np A file on the storage device is in the \textit{open} state from the
beginning of a successful open call to the end of the matching successful close call
on the file handle. It is
erroneous to have the same file \textit{open} simultaneously with two
{\tt upc\_all\_fopen} calls, or with a {\tt upc\_all\_fopen} call and a POSIX/C {\tt open} or
{\tt fopen} call.

\np The user is responsible for ensuring that the file referenced by the {\tt fname}
argument refers to a single UPC-IO file. The actual argument passed on each thread
may be different because the file name spaces may be different on different
threads, but they must all refer to the same logical UPC-IO file.

\np On success, the function returns a pointer to a file handle that can be
used to perform other operations on the file. 

\np{\tt upc\_all\_fopen} provides single-valued errors - if an error occurs, the
function returns {\tt NULL} on ALL threads, and sets {\tt errno} appropriately
to the same value on all threads.

\paragraph{The {\tt upc\_all\_fclose} function}

{\bf Synopsis}
\index{upc\_all\_fclose}
\index{file close}

\npf\vspace{-2.5em}
\begin{verbatim}
    #include <upc.h> 
    #include <upc_io.h> 
    int upc_all_fclose (upc_file_t *fd); 
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fclose} executes an implicit {\tt upc\_all\_fsync} on {\tt fd} and then closes
the file associated with {\tt fd}.

\np The function returns 0 on success. If {\tt fd} is not valid or if an outstanding
asynchronous operation on {\tt fd} has not been completed, the function will return an error.

\np{\tt upc\_all\_fclose} provides single-valued errors - if an error occurs, the
function returns --1 on ALL threads, and sets {\tt errno} appropriately
to the same value on all threads. 

\np After a file has been closed with {\tt upc\_all\_fclose}, the file 
is allowed to be
opened and the data in it can be accessed by using regular C/POSIX I/O calls.

\paragraph{The {\tt upc\_all\_fsync} function}

{\bf Synopsis}
\index{upc\_all\_fsync}
\index{file flush}

\npf\vspace{-2.5em}
 \begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    int upc_all_fsync(upc_file_t *fd);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fsync} ensures that any data that has been written to the
file associated with {\tt fd} but not yet transferred to the storage device is
transferred to the storage device. It also ensures that subsequent file
reads from any thread will see any previously written values (that have not
yet been overwritten).

\np There is an implied barrier immediately before {\tt upc\_all\_fsync} returns.

\np The function returns 0 on success. On error, it returns --1 and sets {\tt errno}
appropriately.

\paragraph{The {\tt upc\_all\_fseek} function}

{\bf Synopsis}
\index{upc\_all\_fseek}
\index{file seek}

\npf\vspace{-2.5em} 

\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h> 
    upc_off_t upc_all_fseek(upc_file_t *fd, upc_off_t offset, 
         int origin);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fseek} sets the current position of the file pointer associated
with {\tt fd}.

\np This offset can be relative to the current position of the file pointer,
to the beginning of the file, or to the end of the file. The offset can be
negative, which allows seeking backwards.

\index{UPC\_SEEK\_SET}
\index{UPC\_SEEK\_CUR}
\index{UPC\_SEEK\_END}
\np The origin parameter can be specified as {\tt UPC\_SEEK\_SET}, {\tt UPC\_SEEK\_CUR},
or {\tt UPC\_SEEK\_END}, respectively, to indicate that the offset must be
computed from the beginning of the file, the current location of the file
pointer, or the end of the file.

\np In the case of a common file pointer, all threads must specify the same
offset and origin. In the case of an individual file pointer, each thread may
specify a different offset and origin.

\np It is allowed to seek past the end of file. It is erroneous to seek to a
negative position in the file. See the Common Constraints number 5 at the
beginning of Section \ref{upc-io-common} for more details.

\np The current position of the file pointer can be determined by calling
{\tt upc\_all\_fseek(fd, 0, UPC\_SEEK\_CUR)}.

\np On success, the function returns the current location of the file pointer
in bytes. If there is an error, it returns --1 and sets {\tt errno} appropriately.

\paragraph{The {\tt upc\_all\_fset\_size} function}

{\bf Synopsis}
\index{upc\_all\_fset\_size}
\index{file size}
\index{end of file}

\npf\vspace{-2.5em} 

\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    int upc_all_fset_size(upc_file_t *fd, upc_off_t size);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fset\_size} executes an implicit {\tt upc\_all\_fsync} on {\tt fd} and
resizes the file associated with {\tt fd}. The file handle must be open for writing.

\np{\tt size} is measured in bytes from the beginning of the file.

\np If {\tt size} is less than the current file size, the file is truncated at the
position defined by {\tt size}. The implementation is free to deallocate file
blocks located beyond this position.

\np If {\tt size} is greater than the current file size, the file size increases to
{\tt size}. Regions of the file that have been previously written are unaffected.
The values of data in the new regions in the file (between the old size and
size) are undefined.

\np If this function truncates a file, it is possible that the individual and
common file pointers may point beyond the end of file. This is allowed and is
equivalent to seeking past the end of file (see the Common Constraints in Section
\ref{upc-io-common} for the semantics of seeking past the end of file).

\np It is unspecified whether and under what conditions this function 
actually allocates file space on the storage device.
Use {\tt upc\_all\_fpreallocate} to force file space to be reserved on the storage device.

\np Calling this function does not affect the individual or common file pointers.

\np The function returns 0 on success. On error, it returns --1 and sets {\tt errno}
appropriately.

\paragraph{The {\tt upc\_all\_fget\_size} function}

{\bf Synopsis}
\index{upc\_all\_fget\_size}
\index{file size}

\npf\vspace{-2.5em} 

\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    upc_off_t upc_all_fget_size(upc_file_t *fd);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fget\_size} returns the current size in bytes of the file
associated with {\tt fd} on success. On error, it returns --1 and sets {\tt errno}
appropriately.

\paragraph{The {\tt upc\_all\_fpreallocate} function}

{\bf Synopsis}
\index{upc\_all\_fpreallocate}
\index{file size}

\npf\vspace{-2.5em}
 \begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    int upc_all_fpreallocate(upc_file_t *fd, upc_off_t size);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fpreallocate} ensures that storage space is allocated for the
first {\tt size} bytes of the file associated with {\tt fd}. The file handle must be open for
writing.

\np Regions of the file that have previously been written are unaffected. For
newly allocated regions of the file, {\tt upc\_all\_fpreallocate} has the
same effect as writing undefined data.

\np If {\tt size} is greater than the current file size, the file size increases to
{\tt size}. If {\tt size} is less than or equal to the current file size, the file size
is unchanged.

\np Calling this function does not affect the individual or common file pointers.

\np The function returns 0 on success. On error, it returns --1 and sets {\tt errno}
appropriately.

\paragraph{The {\tt upc\_all\_fcntl} function}

{\bf Synopsis}
\index{upc\_all\_fcntl}

\npf\vspace{-2.5em}
\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    int upc_all_fcntl(upc_file_t *fd,  int cmd, void *arg);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fcntl} performs one of various miscellaneous operations related
to the file specified by {\tt fd}, as determined by {\tt cmd}. The valid commands {\tt cmd}
and their associated argument {\tt arg} are explained below.

\index{UPC\_GET\_CA\_SEMANTICS}
\begin{description}
\item{\tt UPC\_GET\_CA\_SEMANTICS}
Get the current consistency and atomicity semantics for {\tt fd}. The 
argument {\tt arg} is ignored.
The return value is {\tt UPC\_STRONG\_CA} for strong consistency and atomicity 
semantics and 0 for the default weak consistency and atomicity semantics.
\index{UPC\_SET\_WEAK\_CA\_SEMANTICS}
\item{\tt UPC\_SET\_WEAK\_CA\_SEMANTICS}
Executes an implicit {\tt upc\_all\_fsync} on {\tt fd} and sets {\tt fd} to use the weak 
consistency and atomicity semantics (or leaves the mode unchanged if that mode is already selected).
The argument {\tt arg} is ignored.
The return value is 0 on success. On error, this function returns 
-1 and sets {\tt errno} appropriately.
\index{UPC\_SET\_STRONG\_CA\_SEMANTICS}
\item{\tt UPC\_SET\_STRONG\_CA\_SEMANTICS}
Executes an implicit {\tt upc\_all\_fsync} on {\tt fd} and sets {\tt fd} to use the strong 
consistency and atomicity semantics (or leaves the mode unchanged if that mode is already selected).
The argument {\tt arg} is ignored.
The return value is 0 on success. On error, this function returns 
-1 and sets {\tt errno} appropriately.
\index{file pointer}
\index{common file pointer}
\index{individual file pointer}
\index{UPC\_GET\_FP}
\item{\tt UPC\_GET\_FP}
Get the type of the current file pointer for {\tt fd}.
The argument {\tt arg} is ignored.
The return value is either {\tt UPC\_COMMON\_FP} in case of a common 
file pointer, or {\tt UPC\_INDIVIDUAL\_FP} for individual file pointers.
\index{UPC\_SET\_COMMON\_FP}
\item{\tt UPC\_SET\_COMMON\_FP}
Executes an implicit {\tt upc\_all\_fsync} on {\tt fd}, sets the current file
access pointer mechanism for {\tt fd} to a common file pointer (or leaves it
unchanged if that mode is already selected), and seeks 
to the beginning of the file.
The argument {\tt arg} is ignored.
The return value is 0 on success. On error, this function returns 
-1 and sets {\tt errno} appropriately.
\index{UPC\_SET\_INDIVIDUAL\_FP}
\item{\tt UPC\_SET\_INDIVIDUAL\_FP} 
Executes an implicit {\tt upc\_all\_fsync} on {\tt fd}, sets the current 
file access pointer mechanism for {\tt fd} to an individual file pointer (or
leaves the mode unchanged if that mode is already selected), and seeks to the
beginning of the file.
The argument {\tt arg} is ignored.
The return value is 0 on success. On error, this function returns 
-1 and sets {\tt errno} appropriately.
\index{UPC\_GET\_FL}
\item{\tt UPC\_GET\_FL}
Get all the flags specified during the {\tt upc\_all\_fopen} call for {\tt fd},
as modified by any subsequent mode changes using the {\tt upc\_all\_fcntl(UPC\_SET\_*)} commands. 
The argument {\tt arg} is ignored.
The return value has same format as the {\tt flags} parameter in {\tt upc\_all\_fopen}.
\index{UPC\_GET\_FN}
\item{\tt UPC\_GET\_FN}
Get the file name provided by each thread in the {\tt upc\_all\_fopen} 
call that created {\tt fd}.
The argument {\tt arg} is a valid ({\tt const char**}) pointing to a ({\tt const char*}) location 
in which a pointer to file name will be written.
Writes a ({\tt const char*}) into {\tt *arg} pointing to the filename in implementation-maintained 
read-only memory, which will remain valid until the file handle 
is closed or until the next {\tt upc\_all\_fcntl} call on that file 
handle.
\index{upc\_hint}
\index{file hints}
\index{UPC\_GET\_HINTS}
\item{\tt UPC\_GET\_HINTS}
Retrieve the hints applicable to {\tt fd}.
The argument {\tt arg} is a valid ({\tt const upc\_hint\_t**}) pointing to a ({\tt const upc\_hint\_t*}) 
location in which a pointer to the hints array will be written.
Writes a ({\tt const upc\_hint\_t*}) into {\tt *arg} pointing to an array of {\tt upc\_hint\_t}'s 
in implementation-maintained read-only memory, which will remain 
valid until the file handle is closed or until the next {\tt upc\_all\_fnctl} 
call on that file handle. The number of hints in the array is 
returned by the call.
The hints in the array may be a subset of those specified at 
file open time, if the implementation ignored some unrecognized 
or unsupported hints.
\index{UPC\_SET\_HINT}
\item{\tt UPC\_SET\_HINT}
Executes an implicit {\tt upc\_all\_fsync} on {\tt fd} and sets a new 
hint to {\tt fd}.
The argument {\tt arg} points to one single-valued {\tt upc\_hint\_t} hint 
to be applied. If the given hint key has already been applied to {\tt fd}, 
the current value for that hint is replaced with the provided value.
The return value is 0 on success. On error, this function returns 
-1 and sets {\tt errno} appropriately.
\item
\index{asynchronous I/O}
\index{UPC\_ASYNC\_OUTSTANDING} 
{\tt UPC\_ASYNC\_OUTSTANDING} 
Returns 1 if there is an asynchronous operation outstanding 
on {\tt fd}, or 0 otherwise.
\end{description}

\index{file consistency}
\index{file atomicity}

\np In case of a non valid {\tt fd}, {\tt upc\_all\_fcntl} returns -1 and sets {\tt errno}
appropriately.

\np It \emph{is} allowed to call {\tt upc\_all\_fcntl(UPC\_ASYNC\_OUTSTANDING)} when an
asynchronous operation is outstanding (but it is still disallowed to call
{\tt upc\_all\_fcntl} with any other argument when an asynchronous operation is
outstanding).

\subsubsection{Reading Data}
\label{io-func-read}
\index{file reading}
{\bf Common Constraints}

\npf No function in this section \ref{io-func-read} may be called while an asynchronous
operation is pending on the file handle.

\paragraph{The {\tt upc\_all\_fread\_local} function}

{\bf Synopsis}
\index{upc\_all\_fread\_local}

\npf\vspace{-2.5em}
\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    upc_off_t upc_all_fread_local(upc_file_t *fd, void *buffer, 
         size_t size, size_t nmemb, upc_flag_t flags);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fread\_local} reads data from a file into a local buffer on
each thread\textit{.}

\np This function can be called only if the current file pointer type is an
individual file pointer, and the file handle is open for reading. 

\np{\tt buffer} is a pointer to an array into which data will be read, and each
thread may pass a different value for {\tt buffer}.

\np Each thread reads {\tt (size*nmemb)} bytes of data from the
file at the position indicated by its individual file pointer into the buffer.
Each thread may pass a different value for {\tt size} and {\tt nmemb}. If {\tt
size} or {\tt nmemb} is zero, the {\tt buffer} argument is ignored and that
thread performs no I/O.

\np On success, the function returns the number of bytes read into the local 
buffer of the calling thread, and the individual file pointer of the thread is
incremented by that amount. On error, it returns --1 and sets {\tt errno}
appropriately.

\paragraph{The {\tt upc\_all\_fread\_shared} function}

{\bf Synopsis}
\index{upc\_all\_fread\_shared}

\npf\vspace{-2.5em}
\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    upc_off_t upc_all_fread_shared(upc_file_t *fd, 
         shared void *buffer, size_t blocksize, size_t size,
         size_t nmemb, upc_flag_t flags);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fread\_shared} reads data from a file into a shared buffer in
memory\textit{.}

\np The function can be called when the current file pointer type is either a
common file pointer or an individual file pointer. The file handle must be open
for reading. 

\np{\tt buffer} is a pointer to an array into which data will be read. It must be a
pointer to shared data and may have affinity to any thread. {\tt blocksize} is the
block size of the shared buffer in elements (of {\tt size} bytes each). A {\tt
blocksize} of 0 indicates an indefinite blocking factor.

\np In the case of individual file pointers, the following rules apply: Each
thread may pass a different address for the {\tt buffer} parameter. 
Each thread reads {\tt (size*nmemb)} bytes of data from the
file at the position indicated by its individual file pointer into its buffer.
Each thread may pass a different value for {\tt blocksize}, {\tt size} and {\tt
nmemb}. If {\tt size} or {\tt nmemb} is zero, the {\tt buffer} argument is
ignored and that thread performs no I/O.  On success, the function returns the
number of bytes read by the calling thread, and the individual file pointer of
the thread is incremented by that amount.

\np In the case of a common file pointer, the following rules apply: All
threads must pass the same address for the {\tt buffer} parameter, and the 
same value for {\tt blocksize}, {\tt size} and {\tt nmemb}. The effect is that
{\tt (size*nmemb)} bytes of data are read from the file at the position
indicated by the common file pointer into the buffer.  If {\tt size} or {\tt
nmemb} is zero, the {\tt buffer} argument is ignored and the operation has no
effect.  On success, the function returns the total number of bytes read by all threads,
and the common file pointer is incremented by that amount.

\np If reading with individual file pointers results in overlapping reads into the
shared buffer, the result is determined by whether the file was opened with
the {\tt UPC\_STRONG\_CA} flag or not (see Section \ref{io-func-open}).

\np The function returns --1 on error and sets {\tt errno} appropriately.

\subsubsection{Writing Data}
\label{io-func-write}
\index{file writing}
{\bf Common Constraints}

\npf No function in this section \ref{io-func-write} may be called while an asynchronous
operation is pending on the file handle.

\paragraph{The {\tt upc\_all\_fwrite\_local} function}

{\bf Synopsis}
\index{upc\_all\_fwrite\_local}

\npf\vspace{-2.5em}
\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    upc_off_t upc_all_fwrite_local(upc_file_t *fd, void *buffer,
         size_t size, size_t nmemb, upc_flag_t flags);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fwrite\_local} writes data from a local buffer on each thread
into a file\textit{.}

\np This function can be called only if the current file pointer type is an
individual file pointer, and the file handle is open for writing. 

\np{\tt buffer} is a pointer to an array from which data will be written, and each
thread may pass a different value for {\tt buffer}.

\np Each thread writes {\tt (size*nmemb)} bytes of data from the
buffer to the file at the position indicated by its individual file pointer.
Each thread may pass a different value for {\tt size} and {\tt nmemb}. If {\tt
size} or {\tt nmemb} is zero, the {\tt buffer} argument is ignored and that
thread performs no I/O. 

\np If any of the writes result in overlapping accesses in the file, the
result is determined by the current consistency and atomicity semantics mode in
effect for {\tt fd} (see \ref{io-func-open}).

\np On success, the function returns the number of bytes written by the
calling thread, and the individual file pointer of the thread is incremented by
that amount. On error, it returns --1 and sets {\tt errno} appropriately.

\paragraph{The {\tt upc\_all\_fwrite\_shared} function}

{\bf Synopsis}
\index{upc\_all\_fwrite\_shared}

\npf\vspace{-2.5em}
\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    upc_off_t upc_all_fwrite_shared(upc_file_t *fd, 
         shared void *buffer, size_t blocksize,  size_t size,
         size_t nmemb, upc_flag_t flags);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fwrite\_shared} writes data from a shared buffer in memory to a
file.

\np The function can be called if the current file pointer type is either a
common file pointer or an individual file pointer. The file handle must be open
for writing. 

\np{\tt buffer} is a pointer to an array from which data will be written. It must
be a pointer to shared data and may have affinity to any thread. {\tt blocksize}
is the block size of the shared buffer in elements (of {\tt size} bytes each).
A {\tt blocksize} of 0 indicates an indefinite blocking factor.

\np In the case of individual file pointers, the following rules apply: Each
thread may pass a different address for the {\tt buffer} parameter. 
Each thread writes {\tt (size*nmemb)} bytes of data from its buffer to the
file at the position indicated by its individual file pointer.
Each thread may pass a different value for {\tt blocksize}, {\tt size} and {\tt 
nmemb}. If {\tt size} or {\tt nmemb} is zero, the {\tt buffer} argument is
ignored and that thread performs no I/O.  On success, the function returns the
number of bytes written by the calling thread, and the individual file pointer of
the thread is incremented by that amount.

\np In the case of a common file pointer, the following rules apply: All
threads must pass the same address for the {\tt buffer} parameter, and the 
same value for {\tt blocksize}, {\tt size} and {\tt nmemb}. The effect is that 
{\tt (size*nmemb)} bytes of data are written from the buffer to the file at
the position indicated by the common file pointer.  If {\tt size} or {\tt
nmemb} is zero, the {\tt buffer} argument is ignored and the operation has no
effect.  On success, the function returns the total number of bytes written by all
threads, and the common file pointer is incremented by that amount.

\np If writing with individual file pointers results in overlapping accesses in
the file, the result is determined by the current consistency and atomicity
semantics mode in effect for {\tt fd} (see Section \ref{io-func-open}).

\np The function returns --1 on error and sets {\tt errno} appropriately.

\subsubsection{List I/O}
\label{io-func-list}
\index{list I/O}

{\bf Common Constraints}

\npf List I/O functions take a list of addresses/offsets and corresponding
lengths in memory and file to read from or write to.

\np List I/O functions can be called regardless of whether the current file pointer
type is individual or common.

\np File pointers are not updated as a result of a list I/O read/write
operation.

\index{upc\_local\_memvec}
\index{upc\_shared\_memvec}
\index{upc\_filevec}
\np Types declared in \verb=<upc_io.h>= are

\begin{verbatim}
    struct upc_local_memvec
\end{verbatim}

which contains at least the initial members, in this order:

\begin{verbatim}
    void *baseaddr;
    size_t len;
\end{verbatim}

and is a memory vector element pointing to a contiguous region of
local memory.

\np\vspace{-2.5em}
\begin{verbatim}
    struct upc_shared_memvec
\end{verbatim}

which  contains at least the initial members, in this order:

\begin{verbatim}
    shared void *baseaddr;
    size_t blocksize;
    size_t len;
\end{verbatim}

and is a memory vector element pointing to a blocked region of
shared memory.

\np\vspace{-2.5em}
\begin{verbatim}
    struct upc_filevec
\end{verbatim}

which contains at least the initial members, in this order: 

\begin{verbatim}
    upc_off_t offset;
    size_t len;
\end{verbatim}

and is a  file vector element pointing to a contiguous region of
a file.

For all cases these vector element types specify regions
which are  {\tt len} bytes long.  If {\tt len} is zero, the entry is ignored.
{\tt blocksize} is the block size of the shared buffer in bytes. A {\tt blocksize} of 0
indicates an indefinite blocking factor.

\np The {\tt memvec} argument passed to any list I/O \textit{read} function by a
single thread must not specify overlapping regions in memory.

\np The base addresses passed to {\tt memvec} can be in any order.

\np The {\tt filevec} argument passed to any list I/O \textit{write} function by a
single thread must not specify overlapping regions in the file.

\np The offsets passed in {\tt filevec} must be in monotonically non-decreasing
order.

\np No function in this section (\ref{io-func-list}) may be called while an asynchronous
operation is pending on the file handle.

\np No function in this section (\ref{io-func-list}) implies the presence of barriers at
entry or exit. However, the programmer is advised to use a barrier after
calling {\tt upc\_all\_fread\_list\_shared} to ensure that the entire shared buffer
has been filled up, and similarly, use a barrier before calling
{\tt upc\_all\_fwrite\_list\_shared} to ensure that the entire shared buffer is
up-to-date before being written to the file.

\np For all the list I/O functions, each thread passes an
independent set of memory and file vectors. Passing the same vectors on two
or more threads specifies redundant work.  The file pointer is a single-valued
argument, all other arguments to the list I/O functions are NOT single-valued.

\np EXAMPLE 1:  a collective list I/O read operation. 
The list I/O functions allow the user to specify noncontiguous accesses both
in memory and file in the form of lists of explicit offsets and lengths in
the file and explicit address and lengths in memory. None of the file
pointers are used or updated in this case. 

\begin{verbatim}
#include <upc.h>
#include <upc_io.h>
char buffer[12];
struct upc_local_memvec memvec[2] = {(&buffer[0],4},{&buffer[7],3}};
struct upc_filevec filevec[2];
upc_file_t *fd; 

fd = upc_all_fopen( "file", UPC_RDONLY | UPC_INDIVIDUAL_FP, 0, NULL );
filevec[0].offset = MYTHREAD*5;
filevec[0].len = 2;
filevec[1].offset = 10+MYTHREAD*5; 
filevec[1].len = 5;

upc_all_fread_list_local( fd, 2, &memvec, 2, &filevec, 
                                    UPC_IN_ALLSYNC | UPC_OUT_ALLSYNC);
\end{verbatim}

%\begin{figure}[h]
%\includegraphics[bb=0 0 678 247, width=1\textwidth]{figure4.eps}
%\includegraphics[width=1\textwidth]{figure4}
%\caption{List I/O read of noncontiguous parts of a file to private noncontiguous buffers}
%\end{figure}



\paragraph{The {\tt upc\_all\_fread\_list\_local} function}

{\bf Synopsis}
\index{file reading}
\index{upc\_all\_fread\_list\_local}


\npf\vspace{-2.5em}
\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    upc_off_t upc_all_fread_list_local(upc_file_t *fd,
         size_t memvec_entries, struct upc_local_memvec const *memvec,
         size_t filevec_entries, struct upc_filevec const *filevec,
         upc_flag_t flags);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fread\_list\_local} reads data from a file into local
buffers in memory. The file handle must be open for reading.

\np{\tt memvec\_entries} indicates the number of entries in the array {\tt memvec} and
{\tt filevec\_entries} indicates the number of entries in the array {\tt filevec}. The
values may be 0, in which case the {\tt memvec} or {\tt filevec} argument is ignored and
no locations are specified for I/O.

\np The result is as if data were read in order from the list of locations
specified by {\tt filevec} and placed in memory in the order specified by the list
of locations in {\tt memvec}. The total amount of data specified by {\tt memvec} must
equal the total amount of data specified by {\tt filevec}.

\np On success, the function returns the number of bytes read by the 
calling
thread. On error, it returns --1 and sets {\tt errno} appropriately.

\paragraph{The {\tt upc\_all\_fread\_list\_shared} function}

{\bf Synopsis}
\index{file reading}
\index{upc\_all\_fread\_list\_shared}

\npf\vspace{-2.5em} 

\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    upc_off_t upc_all_fread_list_shared(upc_file_t *fd,
         size_t memvec_entries, struct upc_shared_memvec const *memvec,
         size_t filevec_entries, struct upc_filevec const *filevec,
         upc_flag_t flags);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fread\_list\_shared} reads data from a file
into various locations of a shared buffer in memory. The file handle must be open for reading.

\np{\tt memvec\_entries} indicates the number of entries in the array {\tt memvec} and
{\tt filevec\_entries} indicates the number of entries in the array {\tt filevec}. The
values may be 0, in which case the {\tt memvec} or {\tt filevec} argument is ignored and
no locations are specified for I/O.

\np The result is as if data were read in order from the list of locations
specified by {\tt filevec} and placed in memory in the order specified by the list
of locations in {\tt memvec}. The total amount of data specified by {\tt memvec} must
equal the total amount of data specified by {\tt filevec}.

\np If any of the reads from different threads result in overlapping regions
in memory, the result is determined by the current consistency and atomicity
semantics mode in effect for {\tt fd} (see Section \ref{io-func-open}).

\np On success, the function returns the number of bytes read by the calling
thread. On error, it returns --1 and sets {\tt errno} appropriately.

\paragraph{The {\tt upc\_all\_fwrite\_list\_local} function}

{\bf Synopsis}
\index{file writing}
\index{upc\_all\_fwrite\_list\_local}

\npf\vspace{-2.5em}
\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    upc_off_t upc_all_fwrite_list_local(upc_file_t *fd,
         size_t memvec_entries, struct upc_local_memvec const *memvec,
         size_t filevec_entries, struct upc_filevec const *filevec, 
         upc_flag_t flags);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fwrite\_list\_local} writes data from local buffers in memory to
a file. The file handle must be open for writing.

\np{\tt memvec\_entries} indicates the number of entries in the array {\tt memvec} and
{\tt filevec\_entries} indicates the number of entries in the array {\tt filevec}. The
values may be 0, in which case the {\tt memvec} or {\tt filevec} argument is ignored and
no locations are specified for I/O.

\np The result is as if data were written from memory locations in the order
specified by the list of locations in {\tt memvec} to locations in the file in the
order specified by the list in {\tt filevec}. The total amount of data specified
by {\tt memvec} must equal the total amount of data specified by {\tt filevec}.

\np If any of the writes from different threads result in overlapping accesses
in the file, the result is determined by the current consistency and atomicity
semantics mode in effect for {\tt fd} (see Section \ref{io-func-open}).

\np On success, the function returns the number of bytes written by the
calling thread. On error, it returns --1 and sets {\tt errno} appropriately.

\paragraph{The {\tt upc\_all\_fwrite\_list\_shared} function}

{\bf Synopsis}
\index{file writing}
\index{upc\_all\_fwrite\_list\_shared}

\npf\vspace{-2.5em}
\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    upc_off_t upc_all_fwrite_list_shared(upc_file_t *fd,
         size_t memvec_entries, struct upc_shared_memvec const *memvec,
         size_t filevec_entries, struct upc_filevec const *filevec,
         upc_flag_t flags);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fwrite\_list\_shared} writes data from various locations of a
shared buffer in memory to a file. The file handle must be open for writing.

\np{\tt memvec\_entries} indicates the number of entries in the array {\tt memvec} and
{\tt filevec\_entries} indicates the number of entries in the array {\tt filevec}. The
values may be 0, in which case the {\tt memvec} or {\tt filevec} argument is ignored and
no locations are specified for I/O.

\np The result is as if data were written from memory locations in the order
specified by the list of locations in {\tt memvec} to locations in the file in the
order specified by the list in {\tt filevec}. The total amount of data specified
by {\tt memvec} must equal the total amount of data specified by {\tt filevec}.

\np If any of the writes from different threads result in overlapping accesses
in the file, the result is determined by the current consistency and atomicity
semantics mode in effect for {\tt fd} (see Section \ref{io-func-open}).

\np On success, the function returns the number of bytes written by the
calling thread. On error, it returns --1 and sets {\tt errno} appropriately.

\subsubsection{Asynchronous I/O}
\index{asynchronous I/O}
\index{file reading}
\index{file writing}

{\bf Common Constraints}

\npf Only one asynchronous I/O operation can be outstanding on a UPC-IO file
handle at any time. If an application attempts to initiate a second
asynchronous I/O operation while one is still outstanding on the same file
handle the behavior is undefined -- however, high-quality implementations will
issue a fatal error.

\index{upc\_all\_fwait\_async}
\index{upc\_all\_ftest\_async}
\np For asynchronous read operations, the contents of the destination memory
are undefined until after a successful {\tt upc\_all\_fwait\_async} or \linebreak
{\tt upc\_all\_ftest\_async} on the file handle. For asynchronous write operations,
the source memory may not be safely modified until after a successful
{\tt upc\_all\_fwait\_async} or {\tt upc\_all\_ftest\_async} on the file handle.

\np An implementation is free to block for completion of an operation in the
asynchronous initiation call or in the {\tt upc\_all\_ftest\_async} call (or both).
High-quality implementations are recommended to minimize the amount of time
spent within the asynchronous initiation or {\tt upc\_all\_ftest\_async} call.

\np In the case of list I/O functions, the user may modify or free the lists
after the asynchronous I/O operation has been initiated.

\np The semantics of the flags of type {\tt upc\_flag\_t} when applied
    to the async variants of the fread/fwrite functions should be
    interpreted as follows: constraints that reference entry to a
    function call correspond to entering the fread\_async-
    /fwrite\_async call that initiates the asynchronous operation, and
    constraints that reference returning from a function call
    correspond to returning from the {\tt upc\_all\_fwait\_async()} or
    successful {\tt upc\_all\_ftest\_async()} call that completes the
    asynchronous operation. Also, note that the flags
    which govern an asynchronous operation are passed to the library
    during the asynchronous initiation call.

\paragraph{The {\tt upc\_all\_fread\_local\_async} function}

{\bf Synopsis}
\index{upc\_all\_fread\_local\_async}

\npf\vspace{-2.5em}
\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h> 
    void upc_all_fread_local_async(upc_file_t *fd, void *buffer,
         size_t size, size_t nmemb, upc_flag_t flags);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fread\_local\_async} initiates an asynchronous read from a file
into a local buffer on each thread.

\np The meaning of the parameters and restrictions are the same as for the
blocking function, {\tt upc\_all\_fread\_local}.

\np The status of the initiated asynchronous I/O operation can be retrieved by calling
{\tt upc\_all\_ftest\_async} or {\tt upc\_all\_fwait\_async}.

\paragraph{The {\tt upc\_all\_fread\_shared\_async} function}

{\bf Synopsis}
\index{upc\_all\_fread\_shared\_async}

\npf\vspace{-2.5em}
\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    void upc_all_fread_shared_async(upc_file_t *fd,
         shared void *buffer, size_t blocksize, size_t size,
         size_t nmemb, upc_flag_t flags);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fread\_shared\_async} initiates an asynchronous read from a file
into a shared buffer.

\np The meaning of the parameters and restrictions are the same as for the
blocking function, {\tt upc\_all\_fread\_shared}.

\np The status of the initiated asynchronous I/O operation can be retrieved by calling
{\tt upc\_all\_ftest\_async} or {\tt upc\_all\_fwait\_async}.

\paragraph{The {\tt upc\_all\_fwrite\_local\_async} function}

{\bf Synopsis}
\index{upc\_all\_fwrite\_local\_async}

\npf\vspace{-2.5em}
\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    void upc_all_fwrite_local_async(upc_file_t *fd, void *buffer,
         size_t size, size_t nmemb, upc_flag_t flags);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fwrite\_local\_async} initiates an asynchronous write from a
local buffer on each thread to a file.

\np The meaning of the parameters and restrictions are the same as for the
blocking function, {\tt upc\_all\_fwrite\_local}.

\np The status of the initiated asynchronous I/O operation can be retrieved by calling
{\tt upc\_all\_ftest\_async} or {\tt upc\_all\_fwait\_async}.

\paragraph{The {\tt upc\_all\_fwrite\_shared\_async} function}

{\bf Synopsis}
\index{upc\_all\_fwrite\_shared\_async}

\npf\vspace{-2.5em} 

\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    void upc_all_fwrite_shared_async(upc_file_t *fd,
         shared void *buffer, size_t blocksize,size_t size,
         size_t nmemb, upc_flag_t flags);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fwrite\_shared\_async} initiates an asynchronous write from a
shared buffer to a file.

\np The meaning of the parameters and restrictions are the same as for the
blocking function, {\tt upc\_all\_fwrite\_shared}.

\np The status of the initiated asynchronous I/O operation can be retrieved by calling
{\tt upc\_all\_ftest\_async} or {\tt upc\_all\_fwait\_async}.

\paragraph{The {\tt upc\_all\_fread\_list\_local\_async} function}

{\bf Synopsis}
\index{upc\_all\_fread\_list\_local\_async}

\npf\vspace{-2.5em}
\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h> 
    void upc_all_fread_list_local_async(upc_file_t *fd,
         size_t memvec_entries, struct upc_local_memvec const *memvec,
         size_t filevec_entries,  struct upc_filevec const *filevec,
         upc_flag_t flags);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fread\_list\_local\_async} initiates an asynchronous read of
data from a file into local buffers in memory.

\np The meaning of the parameters and restrictions are the same as for the
blocking function, {\tt upc\_all\_fread\_list\_local}.

\np The status of the initiated asynchronous I/O operation can be retrieved by calling
{\tt upc\_all\_ftest\_async} or {\tt upc\_all\_fwait\_async}.

\paragraph{The {\tt upc\_all\_fread\_list\_shared\_async} function}

{\bf Synopsis}
\index{upc\_all\_fread\_list\_shared\_async}

\npf\vspace{-2.5em}
\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    void upc_all_fread_list_shared_async(upc_file_t *fd, 
         size_t memvec_entries, struct upc_shared_memvec const *memvec,
         size_t filevec_entries,  struct upc_filevec const *filevec,
         upc_flag_t flags);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fread\_list\_shared\_async} initiates an asynchronous read of data from a
file into various locations of a shared buffer in memory.

\np The meaning of the parameters and restrictions are the same as for the
blocking function, {\tt upc\_all\_fread\_list\_shared}.

\np The status of the initiated asynchronous I/O operation can be retrieved by calling
{\tt upc\_all\_ftest\_async} or {\tt upc\_all\_fwait\_async}.

\paragraph{The {\tt upc\_all\_fwrite\_list\_local\_async} function}

{\bf Synopsis}
\index{upc\_all\_fwrite\_list\_local\_async}

\npf\vspace{-2.5em}
\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    void upc_all_fwrite_list_local_async(upc_file_t *fd,
         size_t memvec_entries, struct upc_local_memvec const *memvec,
         size_t filevec_entries, struct upc_filevec const *filevec,
         upc_flag_t flags);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fwrite\_list\_local\_async} initiates an asynchronous write of
data from local buffers in memory to a file.

\np The meaning of the parameters and restrictions are the same as for the
blocking function, {\tt upc\_all\_fwrite\_list\_local}.

\np The status of the initiated asynchronous I/O operation can be retrieved by calling
{\tt upc\_all\_ftest\_async} or {\tt upc\_all\_fwait\_async}.

\paragraph{The {\tt upc\_all\_fwrite\_list\_shared\_async} function}

{\bf Synopsis}
\index{upc\_all\_fwrite\_list\_shared\_async}

\npf\vspace{-2.5em}
\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    void upc_all_fwrite_list_shared_async(upc_file_t *fd,
         size_t memvec_entries, struct upc_shared_memvec const *memvec,
         size_t filevec_entries, struct upc_filevec const *filevec,
         upc_flag_t flags);
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fwrite\_list\_shared\_async} initiates an asynchronous write of
data from various locations of a shared buffer in memory to a file.

\np The meaning of the parameters and restrictions are the same as for the
blocking function, {\tt upc\_all\_fwrite\_list\_shared}.

\np The status of the initiated asynchronous I/O operation can be retrieved by calling
{\tt upc\_all\_ftest\_async} or {\tt upc\_all\_fwait\_async}.

\paragraph{The {\tt upc\_all\_fwait\_async} function}

{\bf Synopsis}
\index{upc\_all\_fwait\_async}

\npf\vspace{-2.5em}
\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    upc_off_t upc_all_fwait_async(upc_file_t *fd)
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_fwait\_async} completes the previously issued asynchronous I/O
operation on the file handle {\tt fd}, blocking if necessary.

\np It is erroneous to call this function if there is no outstanding
asynchronous I/O operation associated with {\tt fd}.

\np On success, the function returns the number of bytes read or written by the
asynchronous I/O operation as specified by the blocking variant of the
function used to initiate the asynchronous operation. On error, it returns
--1 and sets {\tt errno} appropriately, and the outstanding asynchronous operation
(if any) becomes no longer outstanding.

\paragraph{The {\tt upc\_all\_ftest\_async} function}

{\bf Synopsis}
\index{upc\_all\_ftest\_async}

\npf\vspace{-2.5em} 

\begin{verbatim}
    #include <upc.h>
    #include <upc_io.h>
    upc_off_t upc_all_ftest_async(upc_file_t *fd, int *flag)
\end{verbatim}

{\bf Description}

\np{\tt upc\_all\_ftest\_async} tests whether the outstanding asynchronous I/O
operation associated with {\tt fd} has completed.

\np If the operation has completed, the function sets {\tt flag=1} and the
asynchronous operation becomes no longer outstanding;\footnote{%
This implies it is disallowed to call {\tt upc\_all\_fwait\_async} or
{\tt upc\_all\_ftest\_async} immediately after a successful {\tt upc\_all\_ftest\_async}
on that file handle.} otherwise it sets {\tt flag=0}. The same value of {\tt flag} is
set on all threads.

\np If the operation was completed, the function returns the number of bytes
that were read or written as specified by the blocking variant of the
function used to initiate the asynchronous operation. On error, it returns
--1 and sets {\tt errno} appropriately, and sets the {\tt flag=1}, and the outstanding
asynchronous operation (if any) becomes no longer outstanding.

\np It is erroneous to call this function if there is no outstanding
asynchronous I/O operation associated with {\tt fd}.

